System for Highly Available Transaction Recovery for Transaction Processing Systems

ABSTRACT

A highly available transaction recovery service migration system in accordance with one embodiment of the present invention implements a server&#39;s Transaction Recovery Service (TRS) as a migratable service. In one embodiment of the present invention, the TRS is a server instance or software module implemented in JAVA. The TRS migrates to an available server that resides in the same cluster as the failed server. The migrated TRS obtains the TLOG of the failed server, reads the transaction log, and performs transaction recovery on behalf of the failed server. The migration may occur manually or automatically on a migratable services framework. The TRS of the failed server migrates back in a fail back operation once the failed primary server is restarted. Failback operation may occur whether recovery is completed or not. This expedites recovery and improves availability of the failed server thereby preserving the efficiency of the network and other servers.

CLAIM TO PRIORITY

This application is a continuation of U.S. application Ser. No.11/673,727, filed on Feb. 12, 2007, entitled “SYSTEM FOR HIGHLYAVAILABLE TRANSACTION RECOVERY FOR TRANSACTION PROCESSING SYSTEMS”[Atty. Docket No. BEAS-1173US7], which is a continuation of U.S.application Ser. No. 10/341,041 now U.S. Pat. No. 7,178,050, filed onJan. 13, 2003, entitled “SYSTEM FOR HIGHLY AVAILABLE TRANSACTIONRECOVERY FOR TRANSACTION PROCESSING SYSTEMS” [Atty. Docket No.BEAS-1173US2] and to U.S. Provisional Patent Application No. 60/359,226,filed on Feb. 22, 2002, entitled “HIGHLY AVAILABLE TRANSACTION RECOVERYFOR TRANSACTION PROCESSING SYSTEMS”, [Atty. Docket No. BEAS-01173US0],which are incorporated herein by reference.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is related to the following United Statespatents and patent applications, which patents/applications are assignedto the owner of the present invention, and which patents/applicationsare incorporated by reference herein in their entirety:

U.S. patent application Ser. No. 10/366,075, entitled “SYSTEMS ANDMETHODS FOR MIGRATABLE SERVICES”, filed on Feb. 13, 2003, currentlypending [Atty. Docket No. BEAS-01195US2].

U.S. application Ser. No. 10/341,207 now U.S. Pat. No. 7,152,181,entitled “METHOD FOR HIGHLY AVAILABLE TRANSACTION RECOVERY FORTRANSACTION PROCESSING SYSTEMS”, filed on Jan. 13, 2003 [Atty. DocketNo. BEAS-01173US1].

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE INVENTION

Distributed networks are well known to programmers and computer systemarchitects. A distributed network may include multiple nodes, computers,or servers. As used herein, a server is defined as a software process. Anode or cluster is a group of servers that may exist on a singlehardware machine or share a physical resource such as a memory disk.Each server in a network usually has applications or objects thatperform different functions. An application on a particular server maybe initiated by another server or by the server it resides on.Distributed networks are advantageous in that several applicationsrequired to accomplish a task or a process may be distributed amongseveral servers. The distributed applications may then be called uponwhen needed. Processes invoked simultaneously may be run on differentservers instead of weighing down a single server and processor. Thisadvantageously distributes processing power and contributes to a moreefficient network.

Distributed transactions can span multiple servers, and servers oftenhost resource managers (e.g. database connection pools or JMS queues)which participate in distributed transactions. As a result ofdistributed transaction participation, locks or other internal resourcescan be held up in the resource managers (e.g. databases locks areacquired for database records that are updated in a distributedtransaction) on behalf of the distributed transaction until thedistributed transaction is completed. For each distributed transaction,a particular server acts as the coordinator, which drives theparticipating transactional resources to commit atomically and thus thetransaction to completion, via the Two Phase Commit (2PC) protocol. Inthe first phase of the 2PC protocol, the coordinator logs a record ofthe transaction and its participants persistently in its TLOG filesafter all participants are prepared successfully. Once prepared, allparticipants hold on to the acquired locks or other internal resourcesfor the transaction until it is told to commit or rollback by thecoordinator. In the second phase of the 2PC protocol, the coordinatorcommits all the participants, which then make the updates durable andrelease the locks and other internal resources. After all participantsare successfully committed, the coordinator then releases the log recordfrom its TLOG. Thus, if a coordinator fails, all the in-flighttransactions that are logged in its TLOG files cannot be driven tocompletion, and thus all participants cannot release their locks orother internal resources, until the coordinator is restarted. Thus, withsystems of the prior art, transaction recovery cannot take place beforea failed server restarts. This limits the availability of transactionrecovery of the failed server and thus the availability of other XAresources (e.g. JMS backends).

In addition to unexpected server failure, a server may be brought downintentionally. Application servers are often configured to run onspecific machines to service client requests. These machines are broughtdown for periodic maintenance, machine servicing, and other reasons. Asa result, the servers located on the downed machine are not able toservice client requests to that machine or perform recovery of in-doubttransactions until the servers are restarted.

One approach the prior art has taken to address this problem is tomigrate servers and their TLOG files to a back-up or alternate machine.This allows unfinished transactions in a TLOG to be processed thusimproving the availability of the failed server and preserving theoperation and efficiency of a network. One such server migration systemfor use in a distributed network is included in the BEA TUXEDOapplication. TUXEDO supports migration of multiple servers residing on amachine. The servers must either consist of a group of servers or allthe servers that reside on a machine. A group of servers within theTUXEDO application is defined as a collection of servers or services ona machine often associated with a resource manager.

An administrator manually migrates servers using the TUXEDO application.The administrator specifies a primary machine and a secondary or back-upmachine for each group of servers. Once a server group has failed orbeen deactivated by a user, a user may manually migrate the servers fromthe primary machine to the secondary machine. The primary then becomesthe acting secondary machine, and the secondary becomes the actingprimary machine. When the group of servers is to be moved back to theoriginal primary machine, the user shuts-down the back-up machine andthen migrates the server group back to the original primary machine.

Though a TLOG cannot be migrated by itself in Tuxedo, an administratormay manually migrate a TLOG file to a back-up server as a secondary stepto of migrating a server. The TLOG migration is a manual processperformed with tmadmin commands. To migrate a TLOG in TUXEDO, anAtmadmin@ session is started and all servers that write to the TLOG aremanually shut-down by a user. Next, the user dumps the TLOG contentsinto a text file, copies the name of the TLOG file to the back-upmachine, and reads the text file into the existing TLOG for thespecified back-up machine. The user then forces a warm start of theTLOG. Though a user may manually migrate a TLOG in this manner, TUXEDOdoes not support having multiple TLOGs per server.

There are several disadvantages to the prior art such as the TUXEDOapplication. Tuxedo does not support the migration of anything less thana group of servers. Thus, if a single server has crashed in a system orrequires maintenance; multiple servers must be shut-down in order tomigrate the server. Tuxedo requires that all servers that write to aparticular TLOG file must be shut-down while the TLOG file is migrated.Tuxedo also does not support multiple TLOGs residing on a single server.In Tuxedo, there is only one TLOG for a group of servers. Once serversof a machine or group have migrated, and the corresponding TLOG ismigrated thereafter, the secondary machine hosts only the migrated TLOG.Additionally, all migration steps in Tuxedo are done manually, includinga complete shut-down of the secondary server when failing back to theoriginal primary or master server. What is needed is a migration systemthat addresses the deficiencies of existing migration systems such asthe one in Tuxedo.

SUMMARY OF THE INVENTION

A highly available transaction recovery service migration system inaccordance with one embodiment of the present invention implements aserver's Transaction Recovery Service as a migratable service. In oneembodiment of the present invention, the TRS is a server instance orsoftware module implemented in JAVA. Highly available transactionrecovery of a server within a cluster is achieved by migrating the TRSto another available server in the same cluster. This allows the backupserver to read the transaction log and perform recovery on the behalf ofthe failed server. Each server in a cluster has a corresponding TRS,which maintains ownership of the servers's TLOG. When a primary serverfails, the failed servers's TRS migrates to an available secondaryserver that resides in the same cluster as the failed server. Theprimary server and secondary server share access to the same memorydisk. While residing on the secondary server, the migrated TRS obtainsaccess to the TLOG of the failed primary server, reads the transactionlog, and performs transaction recovery on behalf of the failed server.Multiple TRS instances may reside on any server, all of which performingtransaction recovery on a single server. The migration may occurmanually or automatically on a migratable services framework. The TRS ofthe failed primary server migrates back to the primary server in a failback operation once the failed primary server is restarted. Failbackoperation may occur whether recovery is completed or not. No serversneed to be shutdown during TRS failover migration to a secondary serveror during TRS failback migration to the primary server. This expeditesrecovery and improves availability of the failed server therebypreserving the efficiency of the network and other servers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a is a block diagram of a transaction recovery service migrationsystem in accordance with one embodiment of the present invention.

FIG. 1 b is a block diagram of a transaction recovery service migrationsystem after failover in accordance with one embodiment of the presentinvention.

FIG. 2 is a diagram of a flow chart showing manual migration failoveroperation in accordance with one embodiment of the present invention.

FIG. 3 is an illustration of a GUI for implementing the presentinvention in accordance with one embodiment of the present invention.

FIG. 4 is a diagram of a flow chart showing manual migration failbackoperation after recovery is complete in accordance with one embodimentof the present invention.

FIG. 5 is a diagram of a flow chart showing manual migration failbackoperation before recovery is complete in accordance with one embodimentof the present invention.

FIG. 6 is a diagram of a flow chart showing automatic migration failoveroperation in accordance with one embodiment of the present invention.

FIG. 7 is a diagram of a flow chart showing automatic migration failbackoperation after recovery is complete in accordance with one embodimentof the present invention.

FIG. 8 is a diagram of a flow chart showing automatic migration failbackoperation before recovery is done in accordance with one embodiment ofthe present invention.

DETAILED DESCRIPTION

A highly available transaction recovery service migration system inaccordance with one embodiment of the present invention implements aserver's Transaction Recovery Service (TRS) as a migratable service. Inone embodiment of the present invention, the TRS is a server instanceimplemented in JAVA. The TRS migrates to an available server thatresides in the same cluster as the failed server. Highly availabletransaction recovery of a server within a cluster is achieved bymigrating the TRS to another available server in the same cluster. Themigrated TRS obtains the TLOG of the failed server, reads thetransaction log, and performs transaction recovery on behalf of thefailed server. A server may host multiple TRS instances at any time aswell as coordinate their corresponding TLOG transactions. In oneembodiment of the present invention, though a server may host multipleTRS instances and TLOGS, each TRS and TLOG corresponds to only oneserver. The migration may occur manually or automatically on amigratable services framework. The TRS of the failed server migratesback in a fail back operation once the failed primary server isrestarted. Failback operation may occur whether recovery is completed ornot. No servers need to be shutdown during TRS failover migration to asecondary server or during TRS failback migration to the primary server.This expedites recovery of the failed server and while preserving theefficiency of the network and other servers.

A transaction recovery service migration system 100 in accordance withone embodiment of the present invention is shown in FIG. 1 a. System 100includes servers 110, 120 and 140. Each server has a corresponding TRSinstance 112, 122, and 142, respectively. Servers 110 and 120 share acommon disk 130 while server 140 utilizes a separate disk 150. Eachserver has a corresponding transaction log(TLOG) that resides on a disk.Server 110 has TLOG 114 on disk 130, server 120 has TLOG 124 on disk130, and server 140 has TLOG 144 on disk 150. All servers may reside ona single cluster 160. Servers within a cluster may reside on the same ordifferent machines (not shown).

In one embodiment of the present invention, each server is associatedwith only one TLOG. Each TRS has exclusive ownership of the TLOG forit's particular server. Thus, TRS 122 has exclusive ownership of theTLOG for server 120, TLOG 130. When a particular server fails, the TRSfor the failed server may be migrated to an alternate server. Themigrated TRS may then perform recovery on the failed server's TLOG whileresiding on the alternate server. In one embodiment of the presentinvention, a TRS may only be migrated to a server that has access to thesame disk space as the failed server. In particular, the shared diskspace must contain TLOG files for the failed server. In anotherembodiment, an administrator may transfer the TLOG file for the failedserver to the disk that the alternate server can access. The shared diskspace may be a dual-ported SCSI, a storage area network (SAN), or someother reliable shared disk architecture.

For example, if server 110 fails, the TRS 112 can migrate to server 120as shown in FIG. 1 b. Once at server 120, TRS1 112 performs recovery onTLOG 114 corresponding to server 110. In this case, server 110 is theprimary server and server 120 is the back-up, secondary, or alternateserver. A migration of a TRS from a primary server to a secondary serveris called failover. In one embodiment of the present invention, a TRSmay undergo failover migration to a server that shares access to thememory containing the TLOG of the failed server. In FIG. 1 b, TRS 112could not perform recovery on server 110 if migrated to server 140because server 140 and server 110 do not share access to memory 130. IfTRS 112 migrates to server 140, recovery by TRS 112 would require thatserver 140 obtain access to memory disk 130 or experience anothermigration to a server with access to disk 130.

Each TRS is also associated with a migratable target as an alternateserver. In one embodiment, administrators can configure aJTAMigratableTarget element for a clustered server. One example of aJTAMigratableTarget configuration is as follows:

<Server Name=“server1” Cluster=“mycluster” ListenAddress=“campton-1”ListenPort=“7001”> <JTAMigratableTarget Name=“server1”ConstraintedCandidateServers=“server1,server2” /></Server>

The runtime information is available from a JTA runtime MBean:JTARecoveryRuntimeMBean, which can be obtained from a JTARuntimeMBeanMBean. In one embodiment, at least two methods of the JTARuntimeMBeanMBean may provide access to the JTARecoveryRuntimeMBean. One method is:

JTARecoveryRuntimeMBean[ ] getRecoveryRuntimeMBeans( ).

This method returns an array of JTARecoveryRuntimeMBean MBeans thatcorresponds to the TRS instances that are deployed on the currentserver. Another method is:

JTARecoveryRuntimeMBean getRecoveryRuntimeMBean(String serverName).

This method returns the JTARecoveryRuntimeMBean MBean that is associatedwith the specified server. If the corresponding JTARecoveryRuntimeMBeanMBean is not deployed on this server, null is returned. TheJTARecoveryRuntimeMBean MBean has several methods as well. One methodis:

boolean is Active( )

This method returns whether the Transaction Recovery Service iscurrently activated on the server. Another method is:

int getInitialRecoveredTransactionTotalCount( ).

This method returns the total number of transactions that are read fromthe transaction log by the TRS. The administrator may use thisinformation to increase the value of the MaxTransactions attribute ofthe JTAMBean MBean as appropriate. Another method is:

int getRecoveredTransactionCompletionPercent( ).

This method returns the percentage of the recovered transactions thatare completed by the Transaction Recovery Service. In one embodiment,the name of the JTARecoveryRuntimeMBean MBean is the name of theoriginal server of the Transaction Recovery Service.

Though failover and failback migration usually involve moving a singleTRS instance at any time, a server may facilitate multiple TRS instancesresiding on the server and coordinate multiple transactions for TLOGscorresponding to the multiple TRS instances. In this case, the serverperforms recovery for multiple TRS instances in parallel. Server 120 inFIG. 1 b facilitates recovery for failed server 110 as well as its ownrecovery and normal processing. In one embodiment, only the primaryserver may service new transactions. In this embodiment, a back-upserver can not service new transactions for a failed primary server. Toregain its TRS and service new transactions, the failed primary servermust restart and the TRS must migrate back to the primary server.Migration of a TRS from a secondary server back to a primary server iscalled failback. Failback operation may vary according to whetherrecovery for the failed server is completed or not before failbackoccurs. After failback, TRS 112 would again reside in server 110 asshown in FIG. 1 a.

In one embodiment of the present invention, manual migration failover isthe only migration scenario that requires interaction by a user. Anadministrator may manually migrate the TRS of a failed server to anotheravailable server in the same cluster. The operation of a manualmigration failover system in accordance with one embodiment of thepresent invention is shown in block diagram 200 of FIG. 2. Manualmigration failover operation begins at start step 205. A first serverinstance (S1) fails in step 210. This may occur by an act of anadministrator or by server malfunction. In step 220, a user issues amigrate command to trigger the migration of the transaction recoveryservice for the first server instance (TRS1) from S1 to a second serverinstance (S2). This is usually done after the user has discovered afailed server or has shut-down a server. In one embodiment, a user mytrigger the migration of TRS1 from S1 to S2 using a console implementedas a GUI system. The GUI console may be implemented so as to graphicallydisplay different clusters and servers. The user may choose a serverhaving the corresponding TRS to migrate and the back-up server toreceive the TRS. In one embodiment, the migration would be performedusing a Java Transaction API (JTA). A JTA Recovery tab is provided foreach server that allows administrators to specify various attributes ofa Migratable Target and perform manual migration of the TransactionRecovery Service associated with the server. The appearance as viewed ona computer screen of a GUI allowing a user to issue a migrate command inaccordance with one embodiment of the present invention is shown in FIG.3. In another embodiment of the present invention, a user or systemadministrator may trigger a manual migration of a TRS using a commandline. A command line administration tool, implemented as a Java program,may allow a user to specify the TRS to be migrated and what server tomigrate the TRS to. The command line tool may also require a user toenter a username and password in order to perform the migration. Thegeneral format of such a command line command in accordance with oneembodiment of the present invention is shown below.

java weblogic.Admin [-url <url>] [-username <username>] [-password<password>] MIGRATE -jta -migratabletarget <server name> -destination<destination servername> [-sourcedown] [-destinationdown]

In another embodiment of the present invention, manual migration may betriggered by a user programmatically using a JMX MBean. In particular, aJMX MigratableTarget MBean of the TRS may be used to trigger themigration of a TRS from one server to another. An example of the codecomprising a MigratableTarget MBean in accordance with one embodiment ofthe present invention is below.

import weblogic.management.Admin; importweblogic.management.configuration.MigratableTargetMBean; importweblogic.management.configuration.ServerMBean; importweblogic.management.runtime.MigratableServiceCoordinatorRuntimeMBean; //Obtain the MigratableServiceCoordinatorRuntimeMBeanMigratableServiceCoordinatorRuntimeMBean msc = Admin.getAdminServer().getMigratableServiceCoordinatorRuntime( ); // Obtain theMigratableTargetMBean of the server whose Transaction Recovery Serviceneeds to be migrated ServerMBean server1 = (ServerMBean)Admin.getMBeanHome( ).getConfigurationMBean(“server1”,“MigratableTargetConfig”); MigratableTargetMBean mt =server1.getJTAMigratableTarget( ); // Obtain the configurationServerMBean of the server to which the Transaction Recovery Service willbe migrated ServerMBean server2 = (ServerMBean) Admin.getMBeanHome().getConfigurationMBean(“server2”, “MigratableTargetConfig”); // Performthe migration of Transaction Recovery Service of “server1” to “server2”msc.migrateJTA(mt, server2, false /*source up*/, false /*destinationup*/);

Though specific code is listed above, an MBean can be configured andimplemented in various ways to achieve the result of triggering themigration of a TRS from one server to another. These variations of codeare all considered within the scope of the present invention. Thepresent invention is not intended to be limited to the JMXMigratableTarget MBean code example listed above.

Next, the migratable framework detects that the S1 is down in step 230.In one embodiment, the user issued command in step 220 informs themigratable framework that the server is down. When the migratableframework detects the server is down in step 220, the migratableframework moves the TRS to a back-up server. The back-up server may bespecified by a user or be pre-determined by the migratable frameworksystem. After step 230, the migratable framework then activates TRS1 onS2 in step 240. In one embodiment, all migratable services includinginstance TRS1 must implement a particular interface. The interface mustbe registered with the migratable framework and includes migrationactivate and deactivate methods. In this embodiment, migration isactivated when the migratable framework calls the migration activatemethod of TRS1 currently residing on S2. Then, TRS1 reads and processesthe TLOG for S1 in step 250. TRS1 reads S1's TLOG files; instantiatesthe transactions of the TLOG files, puts them into the transaction mapof S2, and schedules resource recovery for S1. As a result, S2 serviceswill read and coordinate the transactions from S1's TLOG. The S2 serverbecomes the coordinator of previously in doubt transactions and talks todifferent coordinators and resource managers to resolve transactions.Next, TRS1 performs recovery on behalf of S1 in step 260 while stillresiding on S2. TRS1 performs recovery on behalf of S1 asynchronously.Meanwhile, the backup server's own transaction manager functions toaccept new transactions and perform its own transaction recovery asusual. Thus, there may be more than one instance of TRS activated on aback-up server at any time, the multiple TRS instances originating fromdifferent servers. The recovery may include driving preparedtransactions to completion and performing resource recovery. Manualmigration failover is then complete and operation ends at step 265.Similar manual migration can be performed to migrate the TRS to anotheravailable backup server if a backup server fails before completing thetransaction recovery actions for the original server.

In one embodiment, failback occurs when a failed primary server restartsand is ready to receive its TRS instance back from a back-up server. Theoperation of a manual migration failback performed after recovery iscompleted in accordance with one embodiment of the present invention isshown in diagram 400 of FIG. 4. System operation begins with start step405. In step 410, an alternate or back-up server S2 completes recoveryfor a primary server S1. In one embodiment, recovery completion occurswhen TRS1 of S1 finishes recovery for S1 while residing on S2. Once TRS1completes recovery, TRS1 relinquishes control of S1's TLOG files. Next,TRS1 migration back to S1 is initiated in step 420. In one embodiment,an administrator may manually initiate migration of the TRS back to theoriginal server. In another embodiment, migration is initiated when TRS1contacts the migratable framework and makes a request to migrate TRS1back to S1. In step 430, the migratable framework completes themigration of TRS1 from S2 back to S1. In one embodiment, the migratableframework first deactivates TRS1 on S2 by calling a deactivation methodof TRS1. During the deactivation of TRS1, S2 performs cleanup andremoves any remaining transactions of S1 from its internal transactionmap. After this deactivation of TRS1, the migratable framework movesTRS1 to S1. Then, the migratable framework activates TRS1 on S1 using acall to an activation method of TRS1. Operation then ends in step 435.When S1 later restarts, S1 will regain ownership of the TLOGcorresponding to S1 and will not need to perform further recovery work.

Manual migration failback may also be performed before recovery iscomplete. Operation of manual migration failback performed beforerecovery is completed in accordance with one embodiment of the presentinvention is shown in diagram 500 of FIG. 5. Operation begins at startstep 505. In step 510, S1 is restarted. Up until just before S1 restart,S2 is still performing recovery work for S1. During S1 startup, S1notifies S2 that S1 is now operational. In one embodiment, thenotification is in the form of an administrative MBean event sent fromS1 to S2. Next, TRS1 migration back to S1 is initiated in step 520. Inone embodiment, TRS1 residing on S2 sends a request to the migratableframework to migrate TRS1 back to S1. Then, TRS1 migrates from S2 to S1in step 530. In one embodiment, an administrator may manually migrateTRS1 back to S1 from S2. This may be performed when the back-up serverfails to implicitly migrate TRS1 back to the original server S1. Duringthis manual migration, the migratable service framework deactivates theTRS1 on S2. The deactivation of TRS1 suspends recovery for S1 and allowsS2 to perform cleanup and remove any remaining transactions of S1 fromits internal transaction map. In another embodiment, the migratableframework first deactivates TRS1 on S2 by calling a deactivation methodof TRS1. The deactivation of TRS1 on S2 suspends recovery processing forS1. Thus, S2 may checkpoint the TLOG for S1, purge transactions in itstransaction map originating from S1's TLOG, and stop resource recoveryperformed for S1. During the deactivation of TRS1, S2 performs cleanupand removes any remaining transactions of S1 from its internaltransaction map. S2 then relinquishes control of S1's TLOG. After thisdeactivation of TRS1, the migratable framework moves TRS1 to S1. Then,the migratable framework activates TRS1 on S1. TRS1 is activated byissuing a call to an activation method of TRS1. Operation then ends instep 545. Though falling under the category of manual migration, noadministrator intervention is required for manual migration failbackbefore recovery is done. Once S1 regains ownership of TRS1, it restartsand completes the remaining transaction recovery work.

Automatic migration occurs without any administrative interventionrequired. Automatic failover and failback migration occur without anyinput from a user. In one embodiment, migration occurs seamlessly andwithout notification to the user. Operation of automatic migrationfailover in accordance with one embodiment of the present invention isshown in diagram 600 of FIG. 6. Operation begins at start step 605.Server failure of S1 occurs at step 610. Next, TRS1 is migrated to S2 instep 620. In one embodiment, TRS1 migration to S2 is triggered when themigratable framework detects the failure of S1. The migratable frameworkthen migrates TRS1 from S1 to S2. In one embodiment, a user may specifya preferred order of back-up servers. A preferred server list asindicated by a user may be stored in the migratable target MBean. Themigratable framework will then attempt to migrate TRS1 to the preferredback-up servers in the order specified by the user. Then, the migratableframework activates TRS1 on S2 in step 630. In one embodiment, TRS1 isactivated when the migratable framework calls a migration activationmethod of TRS1. Next, S1's TLOG is read and processed in step 640. Inone embodiment, during activation on S2, TRS1 reads S1's TLOG filesregarding S1 transactions and configures S2 accordingly. In oneembodiment, TRS1 instantiates the S1 TLOG files, places the files inS2's transaction map, and schedules resource recovery for S1. Thus, S2is configured to be the coordinator of the transactions read from S1'sTLOG. Next, TRS1 performs recovery on behalf of S1 in step 650. In oneembodiment, recovery includes driving prepared transactions tocompletion and performing resource recovery. Automatic migrationfailover operation then ends in step 655. S2's own transaction managerand TRS2 function as usual during automatic migration failover. Similarmanual migration can also be performed to migrate TRS1 to anotheravailable backup server if a backup server S2 fails before completingthe transaction recovery actions for the original server S1.

Automatic migration failback is similar to automatic migration failoverin that no administrative intervention is required. Operation ofautomatic migration failback after recovery is complete in accordancewith one embodiment of the present invention is shown in diagram 700 ofFIG. 7. Operation begins at start step 705. Next, S1 recovery iscompleted in step 710. In one embodiment of the present invention,recovery is completed when TRS1 finishes recovery for S1 while locatedon S2. The TRS1 checkpoints S1's TLOG files and relinquishes control ofS1's TLOG files. Then, TRS1 migration back to S1 is initiated in step720. In one embodiment of the present invention, the migration isinitiated when TRS1 requests the migratable framework to migrate TRS1back to S1. Next, the migratable framework completes migration of theTRS to S1 in step 730. The migratable framework first deactivates TRS1on S2 by calling a deactivation method of TRS1. TRS1 deactivationresults in S2 relinquishing control of S1's TLOG. During thedeactivation of TRS1, S2 performs cleanup and removes any remainingtransactions of S1 from its internal transaction map. After thisdeactivation of TRS1, the migratable framework moves TRS1 to S1. Then,the migratable framework activates TRS1 on S1. TRS1 is activated byissuing a call to an activation method of TRS1. Once migration iscomplete, operation ends in step 735. When S1 is restarted, S1 regainsownership of it's TLOG as a result of TRS1 resides on S1. S1 does notneed to perform additional recovery work.

Automatic migration failback may also occur before recovery of thefailed server is complete. Operation of automatic migration failbackbefore recovery is complete in accordance with one embodiment of thepresent invention is shown in diagram 800 of FIG. 8. Operation beginswith start step 805. Next, S1 is restarted in step 810. At the time ofS1 restart, TRS1 residing on S2 has not completed performing recovery onbehalf of S1. TRS1 migration is then initiated in step 820. In oneembodiment, the migratable framework initiates migration upon detectingthat S1 has performed startup. The migratable framework may detect thefailure of the server itself or be notified of the server startup by anoutside source. In one embodiment, S1 informs S2 that S1 has restarted.After migration has been initiated in step 820, the TRS1 migrates to S1in step 830. In one embodiment, the migratable framework firstdeactivates TRS1 on S2 by calling a deactivation method of TRS1. Thedeactivation of TRS1 on S2 suspends recovery processing for S1 by TRS1on S2. The deactivation includes checkpointing the TLOG for S1, purgingtransactions in its transaction map originating from S1's TLOG, andstopping resource recovery performed for S1. During the deactivation ofTRS1, S2 performs cleanup and removes any remaining transactions of S1from its internal transaction map. S2 then relinquishes control of S1'sTLOG files as TRS1 migrates back to S1. After this deactivation of TRS1,the migratable framework moves TRS1 to S1. Then, the migratableframework activates TRS1 on S1. TRS1 is activated by issuing a call toan activation method of TRS1. Once migration is complete, operation endsin step 835. Once S1 regains ownership of TRS1 and restarts, S1 performsthe remaining recovery work.

A highly available transaction recovery service migration system inaccordance with one embodiment of the present invention implements aserver' Transaction Recovery Service as a migratable service. In oneembodiment of the present invention, the TRS is a server instance orsoftware module implemented in JAVA. Each server in a cluster has acorresponding TRS, which maintains ownership of the servers' TLOG. Whena primary server fails, the failed servers' TRS migrates to an availableback-up server that resides in the same cluster as the failed server.The primary server and back-up server share access to the same memorydisk. While residing on the back-up server, the migrated TRS obtainsaccess to the TLOG of the failed server, reads the transaction log, andperforms transaction recovery on behalf of the failed server. Themigration may occur manually or automatically on a migratable servicesnetwork. Automatic migration requires the TRS be deployed on themigratable service framework. The TRS of the failed server migrates backto the primary server in a fail back operation once the failed primaryserver is restarted. Failback operation may occur whether recovery iscompleted or not. This expedites recovery and improves the availabilityof the failed server thereby preserving the efficiency of the networkand other servers.

Examples of embodiments within the scope and spirit of the presentinvention are included in the Appendix to this patent disclosure.

In addition to an embodiment consisting of specifically designedintegrated circuits or other electronics, the present invention may beconveniently implemented using a conventional general purpose or aspecialized digital computer or microprocessor programmed according tothe teachings of the present disclosure, as will be apparent to thoseskilled in the computer art.

Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art. The invention may also beimplemented by the preparation of application specific integratedcircuits or by interconnecting an appropriate network of conventionalcomponent circuits, as will be readily apparent to those skilled in theart.

The present invention includes a computer program product which is astorage medium (media) having instructions stored thereon/in which canbe used to program a computer to perform any of the processes of thepresent invention. The storage medium can include, but is not limitedto, any type of disk including floppy disks, optical discs, DVD,CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs,EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards,nanosystems (including molecular memory ICs), or any type of media ordevice suitable for storing instructions and/or data.

Stored on any one of the computer readable medium (media), the presentinvention includes software for controlling both the hardware of thegeneral purpose/specialized computer or microprocessor, and for enablingthe computer or microprocessor to interact with a human user or othermechanism utilizing the results of the present invention. Such softwaremay include, but is not limited to, device drivers, operating systems,and user applications. Ultimately, such computer readable media furtherincludes software for implementing Node Managers.

Included in the programming (software) of the general/specializedcomputer or microprocessor are software modules for implementing theteachings of the present invention, including, but not limited to,separating planes of a source image, averaging at least one offoreground and background colors, replacing colors, and compensating forerror introduced by color replacement in one plane by feeding error intoa second plane, storage, communication of results, and reconstructing animage according to the processes of the present invention.

Other features, aspects and objects of the invention can be obtainedfrom a review of the figures and the claims. It is to be understood thatother embodiments of the invention can be developed and fall within thespirit and scope of the invention and claims.

The foregoing description of preferred embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to the practitioner skilled in the art.The embodiments were chosen and described in order to best explain theprinciples of the invention and its practical application, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with various modifications that are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalence.

APPENDIX 1.1 Product Perspective

With systems of the prior art, transaction recovery cannot take placebefore a failed server restarts. This limits the availability oftransaction recovery of the failed server and thus the availability ofother XA resources (e.g. JMS backends). In the present invention, weachieve highly available transaction recovery of a server within acluster by migrating the Transaction Recovery Service to anotheravailable server in the same cluster. This allows the backup server toread the transaction log and perform recovery on the behalf of thefailed server.

Transaction Recovery Service depends on the migratable service frameworkfor manual and automatic migration support. JMS backends, which are XAresources, in turn depend on Transaction Recovery Service migration torecover their resources when they are migrated.

1.2 Product Functions

-   -   Manual migration (both fail-over and fail-back) of Transaction        Recovery Service within cluster    -   Automatic migration (both fail-over and fail-back) of        Transaction Recovery Service within cluster    -   Note that this function is contingent upon automatic migration        support from the migratable service framework.

Configuration, manual migration, and monitoring of Transaction RecoveryService

1.3 User Characteristics and Impact (Usability—Use Cases/UserInteractions)

Administrators can configure, manually migrate and monitor TransactionRecovery Services via either the Administration Console or the JMX API.Manual Migration

2.1 Functional Description

Administrators can manually migrate the Transaction Recovery Service ofa failed server (i.e. the original server) to another available serverin the same cluster. Before the original server restarts, theadministrator may also need to manually migrate the Transaction RecoveryService back to the original server.

Manual migration is recommended if the server has failed and is notexpected to be restarted soon. In the absence of migration, a serverwill perform its own transaction recovery when it is restarted after afailure.

2.2 Functional Requirements 2.2.1 Manual Migration Scenarios

There are two aspects of manual migration: fail-over and fail-back.

2.2.1.1 Fail-Over

When a clustered server (Transaction Coordinator) fails, the TransactionRecovery Service associated with the failed server (i.e. the originalserver) can be manually migrated to another available server (the backupserver) in the same cluster, via either the Administration Console orthe JMX API. Please refer to [5] for description of the JMX API formanual migration. During the migration, the migratable service frameworkactivates the Transaction Recovery Service on the backup server. Duringactivation, the Transaction Recovery Service reads the transaction logof the failed server and initializes the transaction recoveryasynchronously. Meanwhile, the backup server's own transaction managerfunctions (accepting new transactions and performing its own transactionrecovery) as usual. Note that there may be more than one instances ofTransaction Recovery Service (that originates from different servers)activated on a backup server at the same time.

Similar manual migration can also be performed to migrate theTransaction Recovery Service to another available backup server if abackup server fails before completing the transaction recovery actionsfor the original server.

2.2.1.2 Fail-Back

Fail-back happens when migrating the Transaction Recovery Service from abackup server to the original failed server. Note that fail-back isimplicit and does not require administrator intervention unless amigration error occurs, as described below.

2.2.1.2.1 Fail-Back when Recovery is Done

When a backup server finishes transaction recovery before the originalserver restarts, it gives up ownership of the Transaction RecoveryService and migrates it back implicitly to the original server. Noadministrative action is needed in this case. Subsequently, when theoriginal server restarts, it regains ownership of its TransactionRecovery Service. However, it does not need to do further recovery work.

2.2.1.2.2 Fail-Back Before Recovery is Done

When the resources to be recovered are not available, transactionrecovery for the original server may not be finished when the originalserver is to be restarted. In this case, the backup server, on detectingthat the original server is coming up, suspends the ongoing transactionrecovery, performs some internal cleanup, and implicitly migrates theTransaction Recovery Service back to the original server. Noadministrative action is needed in this case. The original server, onceit regains the ownership of its Transaction Recovery Service, restartssuccessfully and finishes the remaining transaction recovery work.

If the backup server fails to implicitly migrate back to the originalserver, the original server will fail to boot. In this case, theadministrator needs to manually migrate the Transaction Recovery Servicefrom the backup server to the original server before rebooting theoriginal server. During the manual migration, the migratable serviceframework deactivates the Transaction Recovery Service on the backupserver. The deactivation suspends the transaction recovery, performssome internal cleanup and gives up ownership of the Transaction RecoveryService. Subsequently, when the original is restarted, it regainsownership of its Transaction Recovery Service and finishes the remainingtransaction recovery work.

Automatic Migration 3.1 Functional Description

Under automatic migration, no administrator intervention is needed, andthe migration of Transaction Recovery Service happens transparentlyunder the control of migratable service framework.

3.2 Functional Requirements 3.2.1 Automatic Migration Scenarios

There are two aspects of automatic migration: fail-over and fail-back.

3.2.1.1 Fail-Over

When the migratable service framework detects that a clustered server(Transaction Coordinator) has failed, it automatically migrates theTransaction Recovery Service associated with the failed server to thenext available server (the backup server) in the preferred server listof the migratable target MBean. During the migration, the migratableservice framework activates the Transaction Recovery Service on thebackup server. During activation, the Transaction Recovery Service readsthe transaction log of the failed server and initializes the transactionrecovery asynchronously. Meanwhile, the backup server's own transactionmanager (including its own transaction recovery) functions as usual.

Similar automatic migration sequences can also happen to migrate theTransaction Recovery Service to another backup server if a backup serverfails before completing the transaction recovery actions.

3.2.1.2 Fail-Back

This happens when migrating the Transaction Recovery Service from abackup server to the original failed server.

There are also two cases:

3.2.1.2.1 Fail-Back without Automatic Migration

This is similar to fail-back when recovery is done for manual migrationas described above in Section 3.2.1.2.1.

When a backup server finishes transaction recovery before the failedserver restarts, it gives up ownership of the Transaction RecoveryService and migrates it back implicitly to the failed server.Subsequently, when the failed server restarts, it regains ownership ofits Transaction Recovery Service. However, it does not need to dofurther recovery work.

3.2.1.2.2 Fail-Back with Automatic Migration

This happens if the backup server has not finished transaction recoverywhen the original server restarts. In this case, the backup server, ondetecting that the original server is coming up, suspends the ongoingtransaction recovery, performs some internal cleanup, and gives up theownership of the Transaction Recovery Service of the original server.The migratable service framework will then transparently migrate theTransaction Recovery Service back to the original server. The originalserver, once it regains the ownership of the Transaction RecoveryService, restarts successfully and finishes the remaining transactionrecovery work.

4. Administrative Requirements

Administrative means will be provided to configure, migrate and monitorTransaction Recovery Services.

4.1.1 Administration/Monitoring/Tuning Requirements (e.g. Console,System Admin)

4.1.1.1 Configuring Transaction Recovery Service

Each server in a cluster is associated with a Transaction RecoveryService, and each Transaction Recovery Service is associated with aMigratable Target. If the Transaction Recovery Service is not configuredfor a particular server, no migration will be enabled. In this case, ifthe server fails, transaction recovery will only be performed after itrestarts.

4.1.1.1.1 Configuring Transaction Recovery Service Via the Console

A new JTA Recovery tab for the server will be provided, from whichadministrators can specify various attributes of the Migratable Targetand perform manual migration of the Transaction Recovery Serviceassociated with the server.

4.1.1.1.2 Configuring Transaction Recovery Service Via config.xml

Administrators can configure a JTAMigratableTarget element for aclustered server. Please refer to [5] for detailed information aboutconfiguring migratable targets in general.

The following is an example of the JTAMigratableTarget configuration:

<Server Name=“server1” Cluster=“mycluster” ListenAddress=“campton-1”ListenPort=“7001”> <JTAMigratableTarget Name=“server1”ConstraintedCandidateServers=“server1,server2” /></Server>

4.1.1.3 Monitoring JTA Recovery Service

Each server maintains runtime information of all Transaction RecoveryService instances that it hosts. The runtime information is availablefrom a new JTA runtime MBean: JTARecoveryRuntimeMBean, which can beobtained from the existing JTARuntimeMBean MBean.

Two new methods are added to the existing JTARuntimeMBean MBean toprovide access to the JTARecoveryRuntimeMBean:

-   -   JTARecoveryRuntimeMBean[ ] getRecoveryRuntimeMBeans( )    -   This method returns an array of JTARecoveryRuntimeMBean MBeans        that corresponds to the Transaction Recovery Service instances        that are deployed on the current server.

JTARecoveryRuntimeMBean getRecoveryRuntimeMBean(String serverName)

-   -   This method returns the JTARecoveryRuntimeMBean MBean that is        associated with the specified server. If the corresponding        JTARecoveryRuntimeMBean MBean is not deployed on this server,        null is returned.

The JTARecoveryRuntimeMBean MBean has the following methods:

-   -   boolean is Active( )    -   This returns whether the Transaction Recovery Service is        currently activated on the server.    -   int getInitialRecoveredTransactionTotalCount( )    -   This returns the total number of transactions that are read from        the transaction log by the Transaction Recovery Service. The        administrator may use this information to increase the value of        the MaxTransactions attribute of the JTAMBean MBean as        appropriate.    -   int getRecoveredTransactionCompletionPercent( )    -   This returns the percentage of the recovered transactions that        are completed by the Transaction Recovery Service.

Note that the name of the JTARecoveryRuntimeMBean MBean is the name ofthe original server of the Transaction Recovery Service.

1. A system for recovering transactions comprising: a primary serverhaving a transaction log, and a backup server, wherein upon a failure ofthe primary server, the backup server is operable to read thetransaction log and perform recovery on behalf of the failed primaryserver.
 2. The system of claim 1, wherein the primary server and backupserver reside in a cluster.
 3. The system of claim 1, whereintransaction recovery is accomplished automatically.
 4. The system ofclaim 1, wherein transaction recovery is accomplished manually.
 5. Thesystem of claim 1, wherein transaction recovery is accomplished manuallyby an administrator.
 6. The system of claim 1, further comprising amigratable framework and a transaction recovery service configured toreceive support from the migratable service framework for manualmigration.
 7. The system of claim 1, further comprising a migratableframework and a transaction recovery service configured to receivesupport from the migratable service framework for automatic migration.8. A cluster of servers configured to provide transaction recoverycomprising: a primary server having a transaction log, the primaryserver and transaction log residing in the cluster; and a backup serverresiding in the cluster and having access to the transaction log, thebackup server operable to read the transaction log and perform recoveryon behalf of the failed primary server.
 9. The cluster of servers ofclaim 8, wherein transaction recovery is accomplished automatically. 10.The cluster of servers of claim 8, wherein transaction recovery isaccomplished manually by an administrator.
 11. The cluster of servers ofclaim 8, further comprising a migratable framework residing in thecluster and a transaction recovery service configured to receive supportfrom the migratable service framework for manual migration.
 12. Thecluster of servers of claim 8, further comprising a migratable frameworkresiding in the cluster and a transaction recovery service configured toreceive support from the migratable service framework for automaticmigration.
 13. A computer readable medium including code to implement: aprimary server having a transaction log, and a backup server, whereinupon a failure of the primary server, the backup server is operable toread the transaction log and perform recovery on behalf of the failedprimary server.
 14. The computer readable medium of claim 13, whereinthe primary server and backup server reside in a cluster.
 15. Thecomputer readable medium of claim 13, wherein transaction recovery isaccomplished automatically.
 16. The computer readable medium of claim13, wherein transaction recovery is accomplished manually.
 17. Thecomputer readable medium of claim 13, wherein transaction recovery isaccomplished manually by an administrator.
 18. The computer readablemedium of claim 13, further comprising a migratable framework and atransaction recovery service configured to receive support from themigratable service framework for manual migration.
 19. The computerreadable medium of claim 13, further comprising a migratable frameworkand a transaction recovery service configured to receive support fromthe migratable service framework for automatic migration.